My primary interest is in brain function. Because the principal job of
the brain is to process information, it is necessary to define exactly
what information is. For that, there is no substitute for Claude
Shannon’s theory of information. This theory is not only quite
remarkable in its own right, but it is essential for telecoms,
computers, machine learning (and understanding brain function).
I have written ten "tutorial introduction" books, on topics which vary
from quantum mechanics to AI.
In a parallel universe, I am still an Associate Professor at the
University of Sheffield, England.
Pierce was a contemporary of Claude Shannon (inventor of information theory), so he learned information theory shortly after it was published in 1949. Pierce writes in an informal style, but does not flinch from presenting the fundamental theorems of information theory. Some would say his style is too wordy, and the ratio of words/equations is certainly very high. Nevertheless, this book provides a solid introduction to information theory. It was originally published in 1961, so it is a little dated in terms of topics covered. However, because it was re-published by Dover in 1981, it is also fairly cheap. Overall, this is a sensible first book to read on information theory.
"Uncommonly good...the most satisfying discussion to be found." — Scientific American. Behind the familiar surfaces of the telephone, radio, and television lies a sophisticated and intriguing body of knowledge known as information theory. This is the theory that has permitted the rapid development of all sorts of communication, from color television to the clear transmission of photographs from the vicinity of Jupiter. Even more revolutionary progress is expected in the future. To give a solid introduction to this burgeoning field, J. R. Pierce has revised his well-received 1961 study of information theory for a second edition. Beginning with the origins…
This is a more comprehensive and mathematically rigorous book than Pierce’s book. For the novice, it should be read-only after first reading Pierce’s more informal text. Due to its vintage, the layout is fairly cramped, but the content is impeccable. At almost 500 pages, it covers a huge amount of material. This was my main reference book on information theory for many years, but it now sits alongside more recent texts, like MacKay’s book (see below). It is also published by Dover, so it is reasonably priced.
Written for an engineering audience, this book has a threefold purpose: (1) to present elements of modern probability theory — discrete, continuous, and stochastic; (2) to present elements of information theory with emphasis on its basic roots in probability theory; and (3) to present elements of coding theory. The emphasis throughout the book is on such basic concepts as sets, the probability measure associated with sets, sample space, random variables, information measure, and capacity. These concepts proceed from set theory to probability theory and then to information and coding theories. No formal prerequisites are required other than the usual undergraduate…
From Side Hustle to Main Hustle to Millionaire
By
Ryan Scribner,
It's time to quit your day job. Anyone with the desire can start a side hustle, and there are more opportunities now than ever before. YouTube personality and personal finance guru Ryan Scribner went from a dead-end job to the limitless potential of self-employment by harnessing the power of the…
This is the modern standard text on information theory. It is both comprehensive and highly technical. The layout is spacey, and the authors make good use of the occasional diagram to explain geometric aspects of information theory. One feature I really like is the set of historical notes and a summary of equations at the end of each chapter.
The latest edition of this classic is updated with new problem sets and material The Second Edition of this fundamental textbook maintains the book's tradition of clear, thought-provoking instruction. Readers are provided once again with an instructive mix of mathematics, physics, statistics, and information theory. All the essential topics in information theory are covered in detail, including entropy, data compression, channel capacity, rate distortion, network information theory, and hypothesis testing. The authors provide readers with a solid understanding of the underlying theory and applications. Problem sets and a telegraphic summary at the end of each chapter further assist readers. The…
This is considered to be a modern classic on information theory. It is a fairly readable text that roams far and wide over many topics. MacKay was extraordinarily clever, curious, and generous (which is why he made the book freely available). His wide range of interests included coding theory and artificial neural networks (machine learning), which occupy a few chapters of the book. In describing the difference between Bayesian and frequentist statistical methods, MacKay pulls no punches, and he can be heard laughing through gritted teeth as he tries to make sense of the nonsensical. This is one of those books in which the author’s many throw-away asides are as enlightening as the main text. The book’s website (below) also has a link to an excellent series of video lectures by MacKay. It is also available for free, just use the direct link below.
Information theory and inference, taught together in this exciting textbook, lie at the heart of many important areas of modern technology - communication, signal processing, data mining, machine learning, pattern recognition, computational neuroscience, bioinformatics and cryptography. The book introduces theory in tandem with applications. Information theory is taught alongside practical communication systems such as arithmetic coding for data compression and sparse-graph codes for error-correction. Inference techniques, including message-passing algorithms, Monte Carlo methods and variational approximations, are developed alongside applications to clustering, convolutional codes, independent component analysis, and neural networks. Uniquely, the book covers state-of-the-art error-correcting codes, including low-density-parity-check codes, turbo…
Few of us take the time to analyze our financial needs and goals to answer that pressing question. In Wealth Odyssey, author Larry R. Frank Sr. uses his extensive financial background to provide a universal road map that will help…
This is really two books, strapped together. The first book by Weaver is an informal introduction to the ideas implicit in the second book by Shannon. Shannon’s book is naturally quite dated in its use of language (e.g. uncertainty is pronounced equivocation), and in its references to ‘current’ technology (e.g. PCM). Despite these caveats, it is still a surprisingly accessible book. Finally, because information theory was developed almost exclusively by Claude Shannon, reading the theory explained in his own words gives some insight into how on Earth he managed to come up with such a radical set of ideas.
Scientific knowledge grows at a phenomenal pace--but few books have had as lasting an impact or played as important a role in our modern world as The Mathematical Theory of Communication, published originally as a paper on communication theory more than fifty years ago. Republished in book form shortly thereafter, it has since gone through four hardcover and sixteen paperback printings. It is a revolutionary work, astounding in its foresight and contemporaneity. The University of Illinois Press is pleased and honored to issue this commemorative reprinting of a classic.
Originally developed by Claude Shannon in the 1940s, information theory laid the foundations for the digital revolution, and is now an essential tool in telecommunications, genetics, linguistics, brain sciences, and deep space communication. In this richly illustrated book, accessible examples are used to introduce information theory in terms of everyday games like ‘20 questions’ before more advanced topics are explored. Online MatLab and Python computer programs provide hands-on experience of information theory in action, and PowerPoint slides give support for teaching.
Written in an informal style, with a comprehensive glossary and tutorial appendices, this text is an ideal primer for novices who wish to learn the essential principles and applications of information theory.
Benghazi: A New History is a look back at the enigmatic 2012 attack on the US mission in Benghazi, Libya, its long-tail causes, and devastating (and largely unexamined) consequences for US domestic politics and foreign policy. It contains information not found elsewhere, and is backed up by 40 pages of…
Everything in nature evolves by trial, error, and success—from fundamental physics, through evolution in biology, to how people learn, think, and decide.
This book presents a way of thinking and realistic knowledge that our formal education shuns. Stepping beyond this ignorance, the book shows how to deal with and even…
